Pin InferenceX and run AgentX with Magpie-managed serving by haofrank · Pull Request #114 · AMD-AGI/Magpie

haofrank · 2026-09-29T05:54:55Z

Magpie can run single-node AgentX benchmarks with SGLang or vLLM, locally or in Docker, without Slurm. Magpie manages server startup, readiness, profiling and cleanup. InferenceX/AIPerf replays agent sessions and measures them; Hyperloom remains responsible for optimization and candidate selection.

Each concurrency point starts a fresh server. Its warmup, measurement and repeated profile captures share that server. Server arguments, environment tuning and verified source changes reach the launched process, with a receipt recording what actually ran.

Example

benchmark:
  framework: sglang
  model: deepseek-ai/DeepSeek-V4-Pro-0813
  precision: fp4
  run_mode: docker
  agentx: enable
  envs:
    CONC: 32
  profiler:
    torch_profiler:
      enabled: true
      num_steps: 20
      num_profiles: 3
      start_seconds: 30
      interval_seconds: 200
      detailed_annotations: false

CONC controls active agent session trees; choose a recipe-supported value. Request lengths come from the trace dataset. This example requests three diagnostic captures of 20 server execution steps each. The initial delay starts after warmup; later intervals start after the previous trace finishes flushing. Capture counts are capped to the measurement window. Disable profiling for canonical performance comparisons: profiled results are diagnostic and cannot supply KEEP evidence.

The example uses basic tracing. Enhanced shape/roofline analysis requires a compatible instrumented runtime and passing trace-quality checks.

Compatibility

New installations pin InferenceX to 408c015; existing checkouts retain their version. Legacy paths and the new inferencex-e2e/ layout both work. The generic packaged throughput client uses an isolated Python 3.12 / Transformers 5 environment; these fixes leave the native AgentX/AIPerf runtime unchanged. The validation harness protects GSM8K's existing evaluation environment with framework constraints and version checks. Built-in tokenizer/trust hooks prevent Hyperloom compatibility preparation from rewriting the audited package.

Validation

Latest Magpie cf80532, paired with Hyperloom 3b6c517, passed focused CPU tests and source/wheel immutability checks. Magpie's full Linux Python 3.10 and 3.12 CI passed.

Canonical GPU baseline/candidate and ProfileExecutor capture integration passed on earlier Hyperloom 80c589d / Magpie 938bfd7. The candidate added --disable-cuda-graph, was slower and was not promoted. Profile captures completed, but embedded selected-rank quality verdicts were blocked/warn: too few step annotations for capture 1, and an unmodelled splitter fallback for capture 2. Certificates were not independently re-read from sidecars. The bypass-only probe forecast was not exercised. Full Hyperloom TraceLens/Roofline analysis was not run. The warning verdict leaves usability unverified, not ruled out.

Latest-head ordinary benchmark + GSM8K remains unverified: the bounded attempt timed out without a PASS artifact or final report, and resource constraints prevented a retry. This does not establish a functional defect. The post-attempt audit found all 87 package files unchanged. Our test containers were removed and all three allocations were cancelled; artifacts were retained. Latest-head native GPU, full Coordinator, KEEP selection and GEAK execution are not claimed. Hyperloom #1675 remains Draft with main conflicts and this unmerged dependency.

Detailed results, warnings and retained artifacts.

Resolve declared InferenceX launchers and persist structured server overrides. Require matching launch evidence before accepting candidate results.

Resolve unregistered AMD single-node SGLang/vLLM models through the declared InferenceX generic capability. Bind model config metadata and context to recipe identity, preserve registered recipe ambiguity checks, and expose verified custom result metadata.

devalshahamd · 2026-10-01T20:30:21Z

@haofrank Thanks for incorporating profiling specific aspects as we discussed. I am reviewing in detail, but posted a comment that might require some changes

haofrank · 2026-10-05T17:09:48Z

Thanks @devalshahamd updated, please review again~

haofrank · 2026-10-05T18:10:00Z

GPU validation completed on 2026-10-05 for Magpie c5c80698fef1b89cc6882264b80d5b306d4e9328.

Hardware/workload: MI355X, GLM-5.2-MXFP4, registered SGLang MTP recipe, TP4/EP4, CONC8. Image: lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260924 (digest sha256:baadaba198e23c46c1dd651b5edefae8000bd5430943bd10b246b563905b3ecf).
Exact clients: InferenceX 408c015be4b22d14c69518643609669405507077, AIPerf 754356e9a39acc6cc6afb242d123bb57c3fb6f75.
Diagnostic AgentX replay: 1200 seconds, 384/384 measured requests successful, 0 errors, 0 cancellations; 15 warmup records excluded.
Profiler: 2 captures × 20 requested steps × 4 ranks = 8 nonempty GPU traces. First capture began 30.043 seconds after the measured phase started. The second began 200.018 seconds after the first capture finished flushing. Both captures are complete, with separate trace directories.
TraceLens generated 30 nonempty CSV reports across the two independent captures. All 145 artifact checks passed, including launch receipt integrity, rank coverage, timing, trace contents and analysis outputs. Both benchmark and verification exited 0.

Scope/limits: this is a real diagnostic replay and basic PyTorch trace test, not a canonical performance baseline or a full Hyperloom optimizer run. Enhanced annotations were not enabled: the stock image lacks SGLang shape_discovery. InferenceX collected power from all 8 visible physical cards although compute used 4; its power validator correctly reported expected_gpu_count_mismatch, so power/energy efficiency is invalid and is not reported. AIPerf also emitted output-length mismatch warnings; those are retained and their cause is not established by the default metric-only export.

haofrank · 2026-10-06T03:57:03Z

AgentX server/benchmark integration and repeated trace capture have been exercised through Hyperloom.

Latest tested heads: Magpie cf80532 / Hyperloom 3b6c517 passed 266 / 295 focused tests. Magpie's Linux Python 3.10/3.12 CI passed. Hyperloom compatibility preparation leaves the audited Magpie package unchanged.
GPU integration: on GLM-5.2-MXFP4 / MI355X / SGLang (TP4, EP4, CONC8), baseline and candidate completed 1184/1184 and 591/591 measured requests with zero measured errors. The candidate was slower; no optimization gain is claimed.
Repeated profiling: completed 1202/1202 measured requests with zero measured errors and produced two captures, eight rank trace files.

GPU results above are from Magpie 938bfd7 / Hyperloom 80c589d, not the latest heads. Output-length warnings mean exact requested lengths remain unverified.

Profiling validates capture only: selected-rank quality checks reported blocked / warn; other ranks received file-inventory checks. Full TraceLens/Roofline analysis has not run, and profiled results cannot supply KEEP evidence. Latest-head native GPU and ordinary benchmark/GSM8K validation remain incomplete, as does the full Hyperloom optimization loop (Coordinator/KEEP/GEAK).

mohbasit · 2026-10-06T23:51:49Z

Tested profiling of workloads with Magpie, overall the profiling seems to be functioning well.

devalshahamd

LGTM, thanks!

haofrank added 3 commits September 28, 2026 20:55

Resolve generic benchmark dependencies before eval changes cwd

a3339dc

feat(agentx): resolve launchers and verify server launch extensions

10052e2

Resolve declared InferenceX launchers and persist structured server overrides. Require matching launch evidence before accepting candidate results.

This was referenced Sep 29, 2026

Withdraw AgentX launcher extensions / 撤回 AgentX 启动器扩展 SemiAnalysisAI/InferenceX#3577

Closed

feat(agentx) Use Magpie native AgentX for Hyperloom optimization AMD-AGI/Hyperloom#1675

Open

Use pinned InferenceX clients with Magpie-owned AgentX serving

f0a24ae

haofrank changed the title ~~Add native AgentX launch overrides and custom model replay~~ Pin InferenceX and run AgentX with Magpie-managed serving Sep 29, 2026

haofrank added 2 commits September 29, 2026 10:59

Keep AgentX acceptance simulation bound to the recipe

7095edb

Merge main after the independent eval source-path fix

70b91fc

haofrank changed the base branch from fix/eval-source-paths to main September 29, 2026 18:04

Show AgentX concurrency explicitly in examples

a8f706c

haofrank had a problem deploying to mi355x-benchmark September 29, 2026 19:53 — with GitHub Actions Error

haofrank requested a review from devalshahamd September 29, 2026 19:54

Add bounded PyTorch profiler capture for managed AgentX

ee1769c

haofrank marked this pull request as ready for review September 30, 2026 20:26

haofrank requested review from a team and sinarafati-amd as code owners September 30, 2026 20:26

haofrank added 4 commits September 30, 2026 13:36

Support repeated AgentX profiler captures with configurable intervals

ecab78f

Cap AgentX profiler captures to the replay measurement window

5590ce6

Fix native AgentX client power defaults and preflight required inputs

b23567b

Map SGLang TP and EP traces for multi-rank TraceLens reports

6585623

devalshahamd reviewed Oct 1, 2026

View reviewed changes

Comment thread Magpie/modes/benchmark/config.py

devalshahamd reviewed Oct 1, 2026

View reviewed changes

Comment thread Magpie/modes/benchmark/agentx_profiling.py

haofrank requested a deployment to mi355x-benchmark October 1, 2026 23:35 — with GitHub Actions Waiting

Support delayed AgentX profiling and framework trace annotations

c5c8069

haofrank requested a review from devalshahamd October 5, 2026 16:33

haofrank added 4 commits October 5, 2026 11:17

Provide workflow defaults for generic InferenceX evaluation

63e3b19

Exclude warmup failures from AgentX measured errors

938bfd7

Match packaged InferenceX client Transformers requirements

311d564

Support client tokenizer options without runtime source patches

cf80532

devalshahamd approved these changes Oct 7, 2026

View reviewed changes

haofrank merged commit 1eb3df6 into main Oct 7, 2026
2 checks passed

Navigation Menu

Repository navigation

Pin InferenceX and run AgentX with Magpie-managed serving - #114

Pin InferenceX and run AgentX with Magpie-managed serving#114
haofrank merged 17 commits into
mainfrom
feature/agentx-launch-extension

haofrank commented Sep 29, 2026 •

edited

Loading

Uh oh!

Uh oh!

devalshahamd commented Oct 1, 2026

Uh oh!

Uh oh!

haofrank commented Oct 5, 2026

Uh oh!

haofrank commented Oct 5, 2026 •

edited

Loading

Uh oh!

haofrank commented Oct 6, 2026 •

edited

Loading

Uh oh!

mohbasit commented Oct 6, 2026

Uh oh!

devalshahamd left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

haofrank commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

devalshahamd commented Oct 1, 2026

Uh oh!

Uh oh!

haofrank commented Oct 5, 2026

Uh oh!

haofrank commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haofrank commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mohbasit commented Oct 6, 2026

Uh oh!

devalshahamd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

haofrank commented Sep 29, 2026 •

edited

Loading

haofrank commented Oct 5, 2026 •

edited

Loading

haofrank commented Oct 6, 2026 •

edited

Loading